Skip to content

Hash FFE shards without allocating - #13490

Draft
leoromanovsky wants to merge 13 commits into
masterfrom
leo.romanovsky/ffe-shard-hash-no-alloc
Draft

leoromanovsky wants to merge 13 commits into
masterfrom
leo.romanovsky/ffe-shard-hash-no-alloc

Conversation

@leoromanovsky

@leoromanovsky leoromanovsky commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

What Does This Do

Shard assignment no longer allocates for each shard. Each distinct salt is hashed once per evaluation. On master, scale/splits drops from 21.2 µs and 33.5 KB to 4.6 µs and 0.8 KB per evaluation.

Assignments do not change. The shard is still int(md5("<salt>-<targetingKey>").hexdigest()[:8], 16) % totalShards, with the UTF-8 bytes that String.getBytes produces, so the result is the same for every salt, key and totalShards.

Motivation

We need to win a performance-sensitive customer, and the scale benchmarks of #13485 (now on master) measure that work. Split-heavy flags were the costliest shape in scale/splits. For every shard of every split it visited, the evaluator did the following:

  • Created a new MessageDigest.
  • Built the string salt + "-" + targetingKey.
  • Formatted the 16-byte digest as a 32-character hex string.
  • Took a substring of it and parsed it with Long.parseLong.

A flag with 100 splits of 2 shards did this up to 200 times per evaluation, although its shards use only 2 distinct salts.

flowchart LR
  E[evaluate] --> S1[split 1 shards] --> H1["new MessageDigest<br/>concat + hex + parse"]
  E --> S2[split 2 shards] --> H2["new MessageDigest<br/>concat + hex + parse"]
  E --> SN[split N shards] --> HN["new MessageDigest<br/>concat + hex + parse"]
  H1 -. same salt, hashed again .- H2
Loading

Additional Notes

Changes

  • ShardHasher (new, package-private) belongs to a single evaluation:
    • It reads the shard from the first 4 digest bytes as an unsigned big-endian int. That is the same value as parsing the first 8 hex characters.
    • It encodes salt, - and the targeting key into a reused buffer and hashes the buffer with a reused MessageDigest. No string or digest object is created for each shard.
    • It caches each salt's hash for the rest of the evaluation. Shards, splits and allocations that share a salt reuse it, whatever their totalShards.
  • FlagEvaluator.evaluate creates the hasher before the first split that has shards, after the TARGETING_KEY_MISSING check. matchesShard takes the hasher. getShard and getMD5Hash are removed.
  • ShardHasherTest (new) checks the new code against a copy of the removed algorithm:
    • 60,000 random salt, key and totalShards triples. The strings mix ASCII, 2-, 3- and 4-byte UTF-8, and unpaired surrogates. totalShards goes up to 2^32−1.
    • Inputs above the buffer-retention limit.
    • Fixed vectors computed with Python's hashlib.
    • The cache: 100 splits that each pair one shared salt with their own salt cost 101 digests (the shared salt is hashed once), a cached hash is reused for each totalShards, and two interleaved evaluations stay independent.
    • The buffer: capacity for 3-byte UTF-8 keys, zeroing after each hash, no retention above 8 KiB, and no table for a single salt. Each of these tests fails when the line it covers is removed or changed.
    • CanonicalFixtureTest and the other module tests pass unchanged.
flowchart LR
  E[evaluate] --> H["ShardHasher (this evaluation)"]
  H --> C{salt cached?}
  C -- yes --> R[reuse hash]
  C -- no --> D["thread's MessageDigest + buffer<br/>first 4 bytes → unsigned int"]
  R --> M["% totalShards → ranges"]
  D --> M
Loading

Scale suite results

ScaleSuiteBenchmark.evaluate on master (5e0370dbbe) and on this PR (f59c5d7756, master merged in), built from the same fixtures (508a342). AWS workspace, Intel Xeon Platinum 8175M, 16 vCPUs, JDK 21.0.10, JMH AverageTime, 1 fork, 3 warm-up and 5 measurement iterations of 2 s, -prof gc. The runs alternated between master and this PR under a shared lock, so no other benchmark ran at the same time. The load stayed around 2, so times carry a few percent of error; allocation is exact.

scale/splits case master This PR Change
All 12 cases 22.4 µs, 33,518 B 5.34 µs, 795 B 4.2× faster, 42× less allocation
0: scale-rollout-allocations (subject-0), 50 allocations with distinct salts 30.4 µs, 43,968 B 15.5 µs, 2,552 B 2.0×, 17×
1: scale-rollout-allocations (subject-38) 798 ns, 1,272 B 423 ns, 408 B 1.9×, 3.1×
2: scale-rollout-allocations (subject-21) 15.2 µs, 24,496 B 7.51 µs, 1,520 B 2.0×, 16×
9: scale-targeted-rollouts (subject-67), rollouts behind a plan rule 824 ns, 1,296 B 476 ns, 408 B 1.7×, 3.2×
10: scale-targeted-rollouts (subject-48) 12.6 µs, 18,928 B 6.61 µs, 976 B 1.9×, 19×
11: scale-targeted-rollouts (subject-0) 12.8 µs, 18,952 B 6.55 µs, 952 B 1.9×, 20×

Cases 0–2 are rollout-only, and cases 9–11 mix targeting rules with rollouts. The scale-many-splits cases (3–5) gain the most, since 100 splits share 2 salts. Measured before the latest master was merged in, those cases went from 1.4–112 µs to 0.8–8.7 µs.

The other case files have no sharded splits on their hot path. Measured against the pre-merge base, which has the same evaluator code: scale/variations, scale/flags and scale/rules keep their allocation (28,067 B, 330 B, 1,904 B), and their times are within noise.

Hashing now allocates one hasher of about 40 B per evaluation that reaches a sharded split. The rest of the 408 B that remain in every case comes from the rest of the evaluation. A flag with one salt keeps it in fields and allocates no table. A flag with more salts allocates a small table: 96 B for 2 to 7 salts, about 2.1 KB for 50.

These numbers come from one benchmark thread, whose digest and buffer are already created. A new thread creates both once (see Limits).

After review, each hash also resets the digest (see Decisions). The numbers above include the reset.

Decisions

  • The digest's first 4 bytes, not the hex string. Big-endian, they are the number that the first 8 hex characters encode, so the result is bit-identical. Other Datadog SDKs compute the shard from the same 4 bytes.

  • A hasher per evaluation, with JDK types in the thread-locals. A per-thread hasher would allocate nothing, but it has three problems:

    • A thread-local that holds an SDK class pins the application's class loader on pooled threads, which leaks it on redeploy.
    • A nested evaluation on the same thread (for example from an EvaluationContext that evaluates a flag) would overwrite the outer evaluation's targeting key.
    • The last targeting key would stay on the heap of every thread.

    The thread-locals hold only the MessageDigest and a byte buffer. After each hash the buffer is zeroed and the digest is reset, and buffers above 8 KiB are not kept. The reset matters: after digest(), the JDK's MD5 keeps the last 64-byte input block, which holds the targeting key, until the next input or reset() (checked on JDK 11, 17 and 21). The cost is the per-evaluation hasher and, for flags with more than one salt, the table above.

  • Cache the hash, not the shard. The hash depends only on the salt and the key, so shards that share a salt but differ in totalShards reuse it.

  • A hand-written UTF-8 encoder. String.getBytes allocates, and a CharsetEncoder needs wrapping buffers. The encoder replaces an unpaired surrogate with ? like getBytes, and the random test compares the two byte for byte.

  • Limits. On virtual threads, or on any thread that evaluates only once, each new thread creates its own MessageDigest and buffer. That is still less than before, when every shard created a MessageDigest. totalShards = 0 still raises the same error as before.

Contributor Checklist

leoromanovsky and others added 4 commits November 17, 2026 00:47
Add ScaleSuiteBenchmark, which evaluates each case of the opt-in scale suite of the shared FFE fixtures. The setup checks every expected value and reason first. Pin ffe-system-test-data to the scale suite PR head.
The scale benchmark sat on the JSON value change and checked JSON
integers by type, so it could not set a baseline on master. It now
compares JSON numbers by value, covers every case file of the scale and
scale-json suites, can isolate one case with the index parameter, and
prints each case's index. A load benchmark measures parsing each suite's
configuration and prints the heap it retains, so changes that move work
to load time show their cost.

The submodule points to the scale-suite fixture branch, which adds the
N x M dimensions and JSON values up to 10 MB.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Compute the shard from the first four MD5 bytes as an unsigned int, which
equals parsing the first eight hex characters, with a per-thread
MessageDigest and encoding buffers instead of a new digest, a hex string
and a concatenated key on each call. Within one evaluation, hash each
distinct salt once, so shards that share a salt across splits and
allocations reuse the hash.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Create one ShardHasher per evaluation instead of reusing one per thread.
A nested evaluation on the same thread can no longer change another
evaluation's targeting key, and a thread keeps no SDK class and no
targeting key: the thread-locals hold only the MessageDigest and the
encoding buffer, which is zeroed after each hash. The first salt is kept
in fields, so a flag with one salt allocates no table.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@leoromanovsky leoromanovsky added type: feature Enhancements and improvements tag: performance Performance related changes tag: ai generated Largely based on code generated by an AI or LLM comp: openfeature OpenFeature labels Nov 17, 2026
@leoromanovsky leoromanovsky added the crucible: reviewing A Crucible review has started; this PR has reviewer attention. label Nov 17, 2026 — with ddtool CLI
@datadog-official

datadog-official Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

🎯 Code Coverage (details)
• Patch Coverage: 95.51%
• Overall Coverage: 83.54% (+22.58%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: f59c5d7 | Docs | Give us feedback!

@leoromanovsky leoromanovsky removed the crucible: reviewing A Crucible review has started; this PR has reviewer attention. label Nov 17, 2026
@dd-octo-sts

dd-octo-sts Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite Status
Startup 🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results
Scenario Candidate master Δ (95% CI of mean)
startup:insecure-bank:iast:Agent 14.01 s 13.98 s [-0.4%; +0.9%] (no difference)
startup:insecure-bank:tracing:Agent 12.88 s 13.04 s [-1.8%; -0.6%] (maybe better)
startup:petclinic:appsec:Agent 17.23 s 17.06 s [+0.1%; +1.8%] (maybe worse)
startup:petclinic:iast:Agent 17.03 s 17.00 s [-0.6%; +1.0%] (no difference)
startup:petclinic:profiling:Agent 16.69 s 16.83 s [-2.0%; +0.4%] (no difference)
startup:petclinic:sca:Agent 17.20 s 16.53 s [-0.3%; +8.5%] (no difference)
startup:petclinic:tracing:Agent 16.18 s 16.24 s [-1.4%; +0.7%] (no difference)

Commit: f59c5d77 · CI Pipeline · Benchmarking Platform UI


Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

leoromanovsky and others added 7 commits November 17, 2026 04:00
The scale benchmarks failed the forbidden API check on System.out, and
nothing outside a manual JMH run checked the scale cases, so a sibling
change that resolves a large configuration wrongly would go unnoticed.

- Add ScaleSuiteTest: evaluate every case of the scale and scale-json
  suites twice, and fail when the benchmark misses a case file. The
  cases, their check and the suite reader move to the test sources and
  the benchmarks reuse them; the reader uses JsonReading.
- Allow System.out in the benchmark setups (@SuppressForbidden).
- Check every case again after the measurement, fail on an error code,
  and reject expectations the check does not handle.
- Rerun the jmh task when the fixtures change.
- Measure retained heap after a first parse, as the smallest of three
  signed samples.
- Describe a row as the unweighted mean over the cases of a file, say
  that hooks are not measured and when to run three forks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The fixture PR now checks that the committed scale files match the
generator and fixes its docs. The case and configuration files are
unchanged, so the benchmark numbers do not change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The fixtures add a case file with ONE_OF lists of the integers 1 to
1000 and of the powers of two up to 2^52, which an index keyed on exact
doubles must still match. Pin them and benchmark the file; its 8 cases
pass on master.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The OpenFeature client passes the provider a new LayeredEvaluationContext
on each call, while the benchmark reused one MutableContext. A rule on
`id` when the context has no `id` reads the layered context's key set,
which it builds again on each call, so the benchmark did not show what
applications pay for it.

evaluateLayered wraps each case's context the way the client does, as
ExperimentationOptOutBenchmark.evaluateFreshLayeredContext does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The fixtures replace the 10 MB JSON flag with a 4 MB one, which brings
the scale-json configuration from 14 MB down to 8 MB. The other cases
are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
After digest() the JDK MD5 engine still holds the last input block,
which contains the targeting key, so a pooled thread kept it. Reset the
digest in the same finally block that zeroes the buffer.

Add tests for the buffer size of three-byte chars, for forgetting the key
and not retaining large buffers, and for creating no table for one salt.
Keep random totalShards within 1 to 2^32-1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Take master's version of every file from #13485, which merged as a
squash commit, so this branch only carries the shard hashing change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp: openfeature OpenFeature tag: ai generated Largely based on code generated by an AI or LLM tag: performance Performance related changes type: feature Enhancements and improvements

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant